The Eclipse Foundation provides individuals and organizations with a commercially focused environment for open source software innovation. It includes git repositories, reviews, issues management, continuous integration, forums and mailing lists among other services. Many well-known and widely used projects are hosted on the forge, including the Eclipse IDE itself, The new Java working group,
This dataset is a dump of all posts sent on all mailing lists hosted at the Eclipse Forge. Although this is public data (the mailing lists can be browsed on the official mailman page) all data has been anonymised to prevent any misuse.
We value privacy and intend to make everything we can to prevent misuse of the dataset. If you think we failed somewhere in the process, please let us know so we can do better.
All personally identifiable information has been scrambled using the data anonymiser Perl module. As a result there is no clear email address in this dataset, nor any UUID or name. However all identical information produces the same encrypted string, which means that one can still identify identical data without knowing what it actually is. As an example email addresses are split (name, company) and encoded separately, which enables one to e.g. identify posters from the same company without knowing the company.
The anonymisation technique used basically encrypts information and then throws away the private key. Please refer to the documentation published on github for more details.
This document is a R Markdown document and is composed of both text (like this one) and dynamically computed information (mostly in the sections below) executed on the data itself. This ensures that the documentation is always synchronised with the data, and serves as a test suite for the dataset.
This dataset is composed of a single big CSV file. Attributes are: list, messageid, subject, sent_at, sender_name, sender_addr, Company.
Examples are provided at the end of this file to demonstrate how to use it in R.
Examples:
| Subject |
|---|
| [tycho-user] Specifying the Target JVM and a possible bug..? |
| Re: [eclipse-mirrors] Ganymede release: June 25 |
| [ptp-dev] Gerrit reminder |
| [rdf4j-dev] Jenkins build is back to normal : rdf4j-tools-develop-verify #40 |
| [tools-pmc] [CQ 9995] Apache jclouds openstack-cinder API: 1.8.0 (ATO CQ8975) |
Main characteristics:
| Sent date |
|---|
| 2009-09-16 19:25:41 |
| 2017-04-17 08:12:01 |
| 2017-12-04 23:30:04 |
| 2010-06-10 15:49:13 |
| 2011-06-24 18:01:35 |
| Sender names |
|---|
| b3KtWhtRQtgTadMN |
| ShM24vD6qlPbFR19 |
| HKmwHIC4dREThJRj |
| NpSw7IvQ+Zn48shH |
| gmPd9gPzeK0KVrxw |
Note: A single name repeated several times will always result in the same scrambled ID. This way it is possible to identify same-author posts without actually knowing the name of the sender.
| Sender addresses |
|---|
| Eeebwh768veO/fWd@hF/B8hhVc0H5XRL1 |
| Eeebwh768veO/fWd@hF/B8hhVc0H5XRL1 |
| H58q5y0lSXIS2Etv@IojoN7A4I0eRTHnT |
| a/Z1KROtT57k7w0B@mFsTHgfjAS6G40Zv |
| IBcO/zwAgEpj7+Kv@hF/B8hhVc0H5XRL1 |
Note: A single email address repeated several times will always result in the same scrambled email address. Furthermore both parts of the email (name, company) are individually scrambled, which means that one can identify email addresses from the same company without actually knowing the real company or name of the sender.
Reading file from ../eclipse_mls_clean.csv.
project.csv <- read.csv(file.in, header=T)
We add a column for the Company, which we extract from the email address (i.e. the domain name):
project.csv$Company <- substr(x = project.csv$sender_addr, 18, 33)
Number of columns in this dataset:
ncol(project.csv)
## [1] 7
Number of entries in this dataset:
nrow(project.csv)
## [1] 352951
Names of columns:
names(project.csv)
## [1] "list" "messageid" "subject" "sent_at" "sender_name"
## [6] "sender_addr" "Company"
The dataset needs to be converted to a xts object. We can use the sent_at attribute as a time index.
require(xts)
project.xts <- xts(x = project.csv, order.by = parse_iso_8601(project.csv$sent_at))
When considering the timeline of the dataset, it can be misleading when there several submissions on a short period of time, compared to sparse time ranges. We’ll use the apply.monthly function from xts to normalise the total number of monthly submissions.
project.monthly <- apply.monthly(x=project.xts$sent_at, FUN=nrow)
autoplot(project.monthly, geom='line') +
theme_minimal() + ylab("Number of posts") + xlab("Time") + ggtitle("Number of monthly posts")
One author can post several emails on the mailing list. Let’s plot the monthly number of distinct authors on the mailing list. For this we need to count the number of unique occurrences of the email address (attribute sender_attr).
count_unique <- function(x) { length(unique(x)) }
project.monthly <- apply.monthly(x=project.xts$sender_addr, FUN=count_unique)
autoplot(project.monthly, geom='line') +
theme_minimal() + ylab("Number of authors") + xlab("Time") + ggtitle("Number of monthly distinct authors")
We want to know what companies posted the most messages in mailing listsacross years. To that end we select the 20 companies that have the larger number of posts and plot the number of messages by company year after year.
comps_list <- head( sort( x = table(project.csv$Company), decreasing = T ), n=20 )
df <- data.frame(Company=character(),
Year=character(),
Posts=integer(),
stringsAsFactors=FALSE)
for (i in seq_along(1:20)) {
project.comp.xts <- project.xts[project.xts$Company == names(comps_list)[[i]],]
project.comp.yearly <- apply.yearly(x=project.comp.xts$Company, FUN=nrow)
for (j in seq_along(1:nrow(project.comp.yearly))) {
year <- format(index(project.comp.yearly)[[j]],"%Y")
comp <- as.data.frame(t(c(names(comps_list)[[i]], year, as.integer(project.comp.yearly[[j]]))))
names(comp) <- c("Company", "Year", "Posts")
df <- rbind(df, comp)
}
}
df$Company <- as.character(df$Company)
df <- df[order(df$Company),]
p <- ggplot(data=df, aes(x=Year, y = Posts, fill = Company)) + geom_bar(stat="identity") +
theme_minimal() + ylab("Number of posts") + xlab('Years') +
ggtitle("Top 20 Companies involved in Eclipse mailing lists across years") +
theme( axis.text.x = element_text(angle=60, size = 7, hjust = 1))
g <- ggplotly(p)
g
#api_create(g, filename = "r-eclipse_mls_companies")